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Educational Infornation Centers are defined as those 
encospassing a broad range of search, retrieval, and disseaination 
activities aiaed at providing both increased and nore effective 
connunication of results of research and developaent in the 
educational field to local educational decision sakers and 
practitioners. Evaluation issues are approached in light of these 
eoiiponentss user needs, inforaation resources (data bases), guestion 
negotiation (search and retrieval), iapact and utilization, and cost 
effectiveness. Centers referred to herein are characterized by 
predoainant reliance on the ERIC (Educational Resources Inforaation 
Center) data base, suppleaented by locally acguired additional 
aaterialsi foraalized interface procedures with their clientele; and 
a decided user'^oriented product concept vherein the eaphasis is on 
the user coaaunity developing the guestions and the inforaation 
service providing responses thereto. A chart illustrating a nusber of 
centers and the subject of their evaluation guestions and the R. I. 
S. E. (research and inforaation services for education) inforaation 
center evaluation fora nith guestions ranging froa topical coverage 
on the product to service concerns, client usage, and knowledge of 
the service are included. (HC) 
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Educational information centers are a relatively new 
phenomenon in the burgeoning area of support services 
for basic education. Visonhaler and Moon (1973), in a 
recent ASIS Annual, identified types of edMcational 
information centers: 

There are two major types of educational Information 
centers: Literature Clearinghouses, and Instructional 
Materials Centers. Presently, hundreds of infonnation 
centers are in operation. The extent and dhrersity of such 
centers is described in various directories such as those by 
Wanger(I972)and Kruzas& Schnitzer(1971). 

Excluding the Instructional Materials Centers, a review 
of the above-cited directories would not reveal the 125 or 
more agencies engaged in computerize bibliographic 
searching of the Educational Resources Information 
Center (ERIC) data base, as reported most recently by 
Embry (1974). Lancaster (1%8), in defining information 
retrieval activities in a broader context, posits some 
useful distinctions: 

Information retrieval is the term conventionally, 
though somewhat inaccurately, applied to the type of 
activity discussed in this paper. An information retrieval 
system does not inform (i.e., change the knowl^ge of) 
the user on the subject of his inquiry. It merely informs 
him on the existence (or nonexistence) and whereabouts 
of documents relating to his request. 

An infonnation retrieval system may retrieve complete 
texts of documents, document surrogates (such as ab- 
stracts), or names and addresses of documents (i.e., full 
bibliographic citations). A system that ultimately 
provides the user with fttll document texts is properly 
called a document retrieval system, whereas a system 
that presents only citations is a reference retrieval 
system. A retrieval system usually operates in several 
stages (e.g., its first output may be in the form of cita« 



tions from which the requester can make a selection). 
Subsequently, the t^quester can ask fliat the complete 
textf of these selected items be presented. AHemath^y, 
the sequence of tiesponses may be (a) document numbers, 
(b) citations, and (c) full texts. 

Even this delin^tion fails to encompass the dis- 
semination activities of state departments of education, 
as revealed in Cutter's (1974) recent review of the oluca* 
tion information activities of nine state education 
agencies. 

The definition of an education information center, as 
h is used in the context of this paper, is one that encom- 
pass^ a broad range of search, retrieval, and dissemina- 
tion activities aim«i at providing both increase and 
more effective communication of the results of research 
and development in the educational field to local educa- 
tional decision makers and practitioners. 

An early pioneer of this type was the MOREL-RIS 
information project, an outgrowth of one of the regional 
ESEA Title IV Research Laboi^tories, which originated 
many of the service concepts that have become common 
to an educational information center (Grimes, 1%9). 
ESEA Title III fiinds provided the seed money for the 
establishment of a number of the other early information 
centers followed by the provision of monies for the sup* 
port of information centers (Lancaster, 1968) serving the 
vocational-technical education community under Re- 
search Coordinating Units. Likewise, the Bureau of Edu- 
cationally Handicapped ftindhig for special education 
information centers came as a result of Title VI of the 
ESEA. It was only in the very late sbrties and the estfly 
seventies that the National Center for Educational Com- 
munication under the leadership of Dr. Lee Burchinal 
and Tom Clemens began to support the notion of 
"comprehensive, one-stop educational information 
centers." 
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At last count, mm^ 25 such ftiUy develop in* 
formatton centers existed natkinaUy« many supported by 
a combination of state and federal funds supplemented 
by service fees. This paper is not an attempt to be 
descriptive of all ajg^ncti^ that have adopts the nomen* 
clature of educational information c^ntetii. Though there 
are great differences in clientele* sources of fwiding« 
nature of services, geographical service area, and 
philosophy of operation, the educational information 
centers referred to herein are characterized by reliance 
cn a predominant data base, (ERIC)« supplemented by 
locally acquire additional materials; formalized inter* 
face proc^ures with their clientele* usually through 
linkage agents or some process of negotiating inquiri^: 
and a decided user*oriented product concept wherehi the 
emphasis is on the user community developing the 
questions and the Information service providing 
responses thereto. While a number of the existing caters 
do provide a selective dissemination of information <SDI) 
and some br iidcasi dissemination activitieSf It is in the 
n'spoNsiw inf^mnatton service components that the 
greatest challenges relating to evaluation and the 
greatest potential for impact on the quality of education 
reside. 

What follows, then, is a brief summary of some of the 
issues that exist in the evaluation of ^ucational in- 
formation centers and an explanation of existing evalua* 
tton strategies and a call for more rigorous analysis and 
evaluation of the services and the impact of these centers* 
It should be made clear, however, that no completely 
common definition of an educational information c^ter 
presently exists: thus comparative data aa a basis for 
evaluation is at present unobtainable. This paper does 
not defend what exists as the most Ideal but deals instead 
with that which is workable and operational given the 
present state*of-the*art. One other limiting characteristic 
of information centers and, therefore, of their evaluation 
is that, in most cases, they are but one of a large number 
of factor, that impact on the decision making and pro- 
gramming of local school district operations. Therefore^ 
the state of affairs presently reveals that the information 
center has little, if any, control over the other variables 
that may impact upon utilization of its services, i.e., 
political, socioeconomic, and other context factors. 

Nevertheless, educational information centers, regard- 
less of their sophistication, have an important if elusive 
role in the improvement of education. As suggested by a 
recent New York State Education Department document 
on the value of educational information ^tems: 

Certainly the most beneficial Improvements In educa* 
tion or in any field come about through the deliberate 
study of alternatives. Rarely can ahematives (ideas, tech- 
niques or programs) be fully adopted in a unique setting. 
Rather, they must be adapted to suit personal administra*' 
tive, organizational or demographic idiosyncrasies. 



Reprdless of the degree or nature oi* adaptation 
required^ ikere mmi be on ^^hm memis Jbr Qrigitmify 
id^fUijyifjS relevant aliematives, be ibey curricubim 
mmeriais or ioiai prc^nmts. Without such a means we 
generally devise c«^ly "add ons" or recreate the work 
accomhiished totally or hi part by others, in short, 
deltbeiate, meaningful ^cational improvement is im* 
pcmsible without such means for alternative identification. 
(Benson, quoted in preface to Reiner < 1 974)]. 

A signal activity, thenu of educational infotmation 
centers is their capacity to provide information about 
alternatives and supporting information for the process 
of adaptating or adopting such ahematives. It is our 
contention that evaluation of the ^ucational Jnforma* 
tion center must proceed from this premise. In counter* 
point to this purpose of supporting the planned change 
or improvemont of education Is the simple role of service 
(providhig information in response to a qu^on). This 
dichotomy of purpose may prove dialecticdl~i.eM future 
educational information centers may to fuUiil both 
roles. Evaluation of education information centers has 
hinged on this dual role vis*a*vis the outcome of their 
functioning. 

While any discu^ion of the components of an educa- 
tional information center is arbitrary, we have decided to 
approach the evaluation issues in light of these com- 
ponents: 

1. User needs 

2. Information resources (data bases) 

3. Question negotiation (seareh and retrieval) 

4. Impact and utilization 

5. Cost e/fecttvcness 

User Needs 

The degree to which user-n^s studio relate to the 
evaluation of information centers is directly related to 
the style of the center in regard to providing broadcast or 
SDI services as opposed to responsive services. It is 
logical that a center that concentrates on responsh^e 
services, in effect, conducts the user*need$ study at the 
time it negotiates t!ic query with each individuaJ client. 
Nevertheless, user-needs studies have had a general value 
in that they have provided a focus for data base acquisi- 
tion considerations and, to some degree, the format of 
information most preferred by various clienteles. Thus, 
the user-needs study may provide the criterion against 
which a gh^en infoiroation center's data base holdings 
may be judged. At the national levd, the federal govern* 
ment has been res(Kinsible for at least one model 
development for an educational information user-needs 
study (Paisley, 1972), A second data base requirements 
study is under way at the present time under contract to 
the National Institute of Education. Hiese efforts have 



indii^ty been tbcus^ on the evaluation of what exhts^ 
paitktilarly in relation to ERIC and Its stibsystems* 
Because all of the educational information centers rely 
heavily on ERIC« the implications of these evaluations 
are signtlScant. In the pHor study^ the methodology was 
also utilised by individual states/infimniation centers* 

Existing infonnation centen have done ^me mer- 
needs analyses on their own; one inference of these 
studies has been that neither information topic nor 
fcMmiat seem to distribute in any predictable way across 
role types in the ^ucational field — e.g., if open spac^ 
elementary education is "high interest,'* this interest 
applies to guidance counselors^ central administrative 
staif* principals* classroom teachers« and Instructional 
support personnel. Likewise* if a recent state mandate 
has required the provision by local districts of a plan to 
handle confidentiality issuer in regard to pupil personnel 
records* model plans, manuals, and •*how-to^o*it** 
strategies are in demand regardl^s of the rote status of 
the local district personnel designated to produce a 
response to the state mandate. More vigorous evaluation 
of this phenomena will* of course, provide ne^ed in- 
sights into the design of SDI services. While constantly 
improving, the planning capacity of practitioners still 
suggests a '*tit^house** res{K)nse to local district problems 
with the result that current user*needs studio seldom 
have particular value in predicting future information 
needs. As a result, many information enters have 
supplemented their user-necds study with conscious 
efforts to monitor federal and state trends in program- 
matic thrusts* futurism activities such as Delphi analysis* 
recommendations of education reform commissions* and 
the monitoring of conference proceedings. Another tech* 
nique that information centers use to anticipate potential 
information needs is the comprehensive acquisition of 
organizational and speciaMnterest group newsletters 
from the national level: Conscientious review of such 
new&letters as well as announcements of research and 
development awards by various f\inding agencies helps 
centers maintain a position at least one jump ahead of 
their clients in regard to potential sources of new in- 
formation. 

In summary, user needs as an aspect of the evaluation 
of the information center is limited except that a hrmd- 
based user-needs study may suggest categories of in- 
formation desired by present nonusers and may provide 
gross measures of the adequacy of information resources. 



inforoiatlon Resources (Data Bases) 

For mmt educational information centers, the main and 
most commonly searched data base Is ERIC. Subfiles of 
ERIC such as AIM and ARM. the compilations from the 
Vocational and Technical Education Information Center 
at Ohio State University, and the EC collection from the 



Council for Exceptional Children form a smmdary 
resource tor information center personnel* Several 
hundred indexing and abstracting resources and jour- 
nals in the educational field keep the literature searcher 
abreast of the latest written materials^ 

Evaluations of the ERIC system are fed back to 
Central ERIC through {^rsonal communications and 
semiannual conferences of data base users. Educational 
information center personnel conscientiously provide 
this type of ftE^back« rabe questions about the ERIC 
system* work to Improve it through such methods as the 
VocalHilary Review Panel, provide material for its 
acquisiticm fUnction, and have continuously supported 
this nationallE^ resource. 

A technique Implement^ to me other raiources of an 
information center is the citation count. A manual 
counting of the bibliographic citations of journals and 
indexes found in completed searches will yield the 
number of times a particular resource has been utiliased* 
Results of this count may be used in decisions for sub- 
scribing or ^^ordering specific materials. A more in- 
formal evaluation of the journals may be through staff 
discussions or individual preferences expressed to the 
acquisition component of the center. 

When self^leveloped or new data bas^ are considered 
for the information resources of a center, the criteria for 
adoption are a supphment not a supplanting of 
materials to the ERIC system* a compatible ^formm such 
as descriptor assignment to documents* and a machine 
readable data base for ease in searching. Some facilities 
hpve their own resources indexed in ERIC terms. 
Examples of these are Research and Information Ser- 
vice? for Education (R.I.S.E.) and the Educational 
Resource Center of Montgomery County* Maryland, 
Schools. 

Question Negotiation (Search and Retrieval) 

Although the question of what constitutes a search may 
never be fully determined* the literature search function 
has basic processes; commonalities of practice may be 
observed from center to center (see Figure 1). Each 
facility, however, will differ in its information resources, 
its reliance on manual and/or computer searching, and 
the constraints under which it operates* i.e.~its charter 
sponsor, department of education, etc. 

Evaluation issues addressed in the search and retrieval 
operations of a center are those of information resources, 
literature search and support staff such as the library, 
and internal feedback mechanisms on the search opera- 
tions and the products. In regard to information 
resources, a center must determine* within its dollar con- 
siderations and facility situation* whether to contract 
with one or both of the two large computer service 
systems to broaden their searching capacity by access to 
four or five major educational data bases on line. Several 
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inforniatton centers with on*&ite computers have access 
to their own specific collections as well An analysis of 
these centers illustrates a variety of products as well as 
capabilities. When users evaluate the service of a center, 
it is specific to that center since users do not usually 
contact more than one center for information needs. In 
time, a comparisi>n of center to center may occur as more 
ser\ice facilities with open search policies, which allow 
clients from outside a specific geographic area or 
clientele group to request service, move Into a circum- 
scribed locale. 

Stalf evaluation has been concerned with performance 
in negotiating questions and conducting searches of the 
literature. An area that needs more investigation is the 
training of the personnel who perform these functions. It 
should be noted that until recently, very few resources 
addressed the training of searchers or support personnel. 
Reference librarians have observed that while theory on 
the topics of question negotiation and searching is avail- 
able, little if any, preservice simulations or hands-on 
activities are presently conducted. On-the-job training, 
an apprenticeship of sorts, has been the only way to gain 
this type of experience. As a result, searching often 
suffered from trial and error tactics to gain information. 
As clients liecome more sophisticated in their usage of 
the service, unskilled efforts will not be tolerate. 

Another issue for consideration in staff evaluation is 
how much time and effort shiHild be placed on the review 
of a product by supervisory staff before it is sent to the 
client. Constraints of a small staff and high demand for 



information may cau^ a center to fon^o this type of 
review. At others it may be a mandated pt^icedure, 

intettial mechanisms (w rating the efficiency of srarch 
and retrieval operations are based on the premise that 
the data base that yields the mcmt relevant information in 
the least amiuini of search time is the one to Investi^te 
first. This cannot always be predetermmed, but 
familiarity and experience with the tirsoufces aids In con- 
structing a resource attadc strategy. A ti^nique used by 
R.I.S.E, is to require the literatutis searcher to report, on 
a search strat^ work sheet housed in the final product, 
the usefulness of each data base, indexing and abstract- 
ing resource, and outside aj^ncy us^ in i^searchifig the 
topic. This evaluation is considered when the informa- 
tion packet, the search, is updated at a later time. 

Another form of process evaluation on retrieval may 
be conducted by an outside consultant with expertise in a 
particular field. This expert may called in to review a 
series of Information packets for emumt comprehensive* 
ness and quality of matenal. He or she may also suggest 
deletions or inclusions of <^rtain material and may be 
acquainted whh human resources and authorities not 
publish^ in journals indexed by the major systems. The 
expert's evaluation should b .*onfined to the items above 
and not applied to the a{^/ \ « iriateneiim of the material to 
the client. This determin«i; on is best made by the 
searcher in conjunction with ^ impetent negotiator. 

Because the tendency on tht part of the client is to rate 
only the actual product re<\*ived, some often reply *i 
didn't get much material on this question,'' not con- 
sidering the time and eflbrt it took to retrieve the infor- 
mation they did get. Rarely does the reply come back, 
even with weighty tomes of material, *•! receh^ed too 
much Information on the topic.** It seems difiicult to 
convince the user that the center is providing a service, 
not guaranteeing reams of information on a topic. Most 
educational information centers have designed or 
borrowed from others those elements of a service 
evaluation that best fit their particular situations and 
forms of output. Questions on the evaluation form 
(which in some cases accompani<^ the product while in 
others follows the packet by a time^delay period of six 
weeks) channel the client to rate not only the content 
materials but also the service features of the center. A 
chart illustrating a number of centers and the subject of 
their evaluation questions is found in Figure 2. Figure 3 
depicts the R.I.S.E. information center evaluation form, 
last revise in 1970. which has questions ranging from 
topical coverage on the product to service concerns, 
client usage, and knowledge of the service. 

The evaluation form asks for identification and role 
information: anonymity is not desired hcve since replies 
on the evaluation may warrant a reworking of the search 
strategy or inclusion of additional material to satisfy a 
user query. 
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TIS ' Texas Information Service 

Rl - EIC - Rhode Island - Educational Information Center 

INFORMS - Iowa Network for Obtaining Resource Materials for Schools 

RISE - Research and Information Services for Education (Pennsylvania) 

F ERIC - Florida Educational Resources Information Center 

TENN RCU - Tennessee Research Coordinating Unit 

ACES - Area Cooperative Educational Services (Connecticut) 

NY EPS IS - New York Educational Programs and Studies Information Service 

NIE - ERC - National Institute of Education - Educational Reference Center 
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RESEARCH ANO INFORMAT»ON SERVICES FOR BOWCATtON 
t9SAt.t.eNOAl.EROAO KINO Of PRUSSIA, PA. 19406 



Search # 



NAME J ^ 
f>OSITiON 



SCHOOL OR tNSTITUTtON^ 
SEARCH TITLfi 



Olrecttons; Check only one altefnattve for each question. Most of the questions 
allow you toBveck one of five alternatives frcm "very gooil" to 
"nesllglble". Please return In the enclosed self-addressed 
envelope at your earliest convenience* 

VERY COOP A80VE AVERAGE AVERAGE BELOW AVERAGE WEGLtCIBLE 

▼ ▼ ▼ ▼ ▼ 

1. To what extent did you find it easy to specify your topic with the person 
handling your request? 

o □ □ □ □ 

2. To what extent was your topic adequately covered by the R.i.S.E. 
information package? 

o o a o a 

3. To what extent did the infoimation sent by R.I.S.E. increase your understanding 
of the requested topic? 

a a o □ a 

k. To what extent do you consider the literature searehfng service (in general) 
of R.I.S.E. to be useful for your professional functioning? 

o o O 0 □ 

5. Which part of the infonnation package do you consider most valuable? 

a copies of documents C3 R.I.S.E. bibliography 

6. You would rate the copies of documents that were sent to you as: 

go too technical 

technical, but useful 
Q well-balanced between technical and non-technical 

g non-technical , but useful 

*^ too non- technical 



7. The Information sent to you arrived: 

8 much earlier than expected ^ 
slightly earlier than expected 
Q in time for your purposes 
□ late but still useful 
O too late for your purposes 

8. Your original Intent (check only the single most pertinent Intent) for 
the information package was; 

Q making a decision concerning an educational issue 

O planning a program that currently Is not available 

0 modifying or Improving a program that currently exists 

Q increasing professional background knowledge about a topic 

O other (please specify) 

9. Do you think you could have done as good a r*-erature search as the one 
received from R.I.S.E. if given ample tiffle** 

O yes D no 

10. Did you look up any of the citations on the bibliography for which no 
copies of the document were provided? 

O yes O no 

11. What part of the literature search would you have done alone if the service 
had not been available from R.I.S.E.? 

U all a little 

□ most Q none 

O some 

12. After receiving the R.I.S.E. Information package, how was It actually used 
(please use other side If necessary)? 



13, What general comments, criticisms, suggestions, etc., can you offer about 
the literature search (please use other side If necessary)? 



H. How did you find out about the services of R.I.S.E.? 

□ newsletter □ R.I.S.E. brochure 

0 newspaper D R.I.S.E. clients 

O In-service O other means (specify), 

a RUS 



THANK YOU FOR YOUR COOPERATION. WE WILL USE YOUR FEEDBACK TO IMPROVE THE 
SERVICES OF R.I.S.E. 



Thi? uH'r then mki^ a %tfi^ of t^r qtiesttons tbr 
which he w %h^: h to chmssfi only otie answer oti a Ukert* 
typi? icttte ranging from very gtnHl to iH^ligibte. The first 
questbn in the series ablates to the negotiation of the 
client s pn>bh?m or need area and retiecis the enpertise of 
the center personnel in malcing the cHent feel comtbrt* 
able and at ease while {muring to find out wimt the cUem 
is ii?dHy seeking. In cases where extension agents or 
intermediaries arc for the neg^lation {x>rtion of the 
service, the answers to this question prt»vide feedtiack on 
the perfcinnance of these Indtviduafs^ 

Question 2 requests a response on the data base as wdl 
as service. The topic may be inadequately covered 
becaujie no such Informa^on exists, or it may be that the 
searcher UH*d a faulty strategy or failed to investigate the 
right MHirces. Clients otlien assume that, because they 
have questionib^ an ansi^^r resides somewhere^ However, 
it may not reside where it can be access^ or t^trieved. As 
satisfied clients become more si^phisticated, In repeat 
uses of the ^er>ice. thel« questions are better defined and 
their espectatii>ns are more reasonable. 

The thin! query on the form refers to the amount of 
new information received by the client through the 
pnKtuct. Many wilt note that their understanding was 
increased by the effort, but they were satisfi^ with 
the information because it reinforced a position or stand 
ihcy could then justify to a decislon*making body. An 
esampie might be the introduction of a language* 
experience approach to reading in the curriculum 
bccauw student attitudes toward reading change more 
piisitively. '{lie client may have been aware of both the 
program and the change in student attitude but needai 
the literature to per!^uade the school district to adopt the 
concept. 

I1)e last question in this series ^ekii to determine the 
client^ altitude regarding the value of information 
ser%ices in general. Even in cases where the client was 
dissatisfied with the specific output, his or her positive 
answer indicates improved information-seeking behavior 
through forced-choice responses. 

in tile second series of questions, the client is asketH to 
give simie detail on the content of the product, a ratinj, of 
the service, and perceived and actual usage of the 
materials. 

It is c^ften difiicult for R.IS.E clients to separate the 
value the di>cuments from the bibliography. Many 
insen tlieirown answer: both. Traditionally R.ME has 
provided reference and document retrieval. Therefore. It 
is likely they want the documents. Once served the full 
citurse. having only the appetii^r would not be suHicient. 

The rating of the documents in question 6 is related to 
thi» client's background and not necessarily bis other role 
or le%el of learning. For example, identical copies of a 
search cm •*Camping in the Elementary Grades'' sub- 
mitted to a teacher and principal In the same school were 



rated as nontechnical bui a^^fut and it^haicaL hut 
useful r^peetively. Most clients rate the productei as well 
balanced. 

The service feature of timeliness is rated in the next 
question. In question negotiationt a critical date is deter* 
mhied by the cHent and Intbrmation centi^ personnel. 
The iiteratui^ searcher tries to meet the time demami 
but may be thwarted by outside resmir^ reqi^^ not 
received hi time to meet tt» diet's needs. As eltents use 
the service more* they increai^ the le^ time and are lei»i 
likely to make 'i need h yesterday" demands. 

Questbn 8 begs the intention of the request. Although 
a single reply is desired, the intention may meet several 
of the alternative ansi^ers. This question hi compared 
with question 12 to if {^m?ehr^ atid actual needs 
have common ground. It may be that, al^er the Infor* 
mation was received, h was um^ for an entirely difi*erent 
purpose. 

Th^ next series of questions, 9*1 1 , ask the client about 
his or her own reseati?h and retrieval efforts. Mat y 
respond that they could do a ^aod search but do not have 
the luxury of time. For that same reason, many eltents do 
not go beyond the packet to retrieve additiotial docu* 
ments. Time and resouitfes are given as reasons many 
clients would attempt only a small search effort if this 
type of service wei^ not available. 

Questions 12 and 13 of^en elicit long narratives thsm 
the clients explaining their actual use of the product and 
its dissemination among other statf* members. Ciitichmis 
and suggestions are of^en elaborations on questions with 
single response or forc^^^hoice answers, Rrst^time 
users have no benchmarks* As clients reuse the service, 
they compare U with previous experience, and theU* 
responses become more critical and constructive. 

The last question attempts to find out how the client 
came to use the service and If it is a repeat usage. Many 
times a client checks more than one response indlcathig 
multiple iniluenees on his or her decision to the 
service. 

An 80 percent return rate indicates the value of this 
evaluation strategy to both the client and the information 
wnter. Afller i^view by the appit^prlate literature 
searcher and her supervisor, decisions are made to 
m<Klify the search or keep it In its present completed 
status. One copy of the form is placed in the seattrh so 
subsequent users can see the ratuig applied by the 
original requester. A second copy is sent to the finking 
agent representing the requester's geographk? area to 
provide feedback on his or her role as negotiator 
(question 1 ) and to provide the agent with an overall 
evaluation of R.hS.E, services. 

The evaluation form in Figure 3 is sent only to clients 
requestin , a comprehensive Iherature «^arch. Figure 4 
illustrates the evaluation form sent to recipients of a 
computer-generated bibliogi^phy--a quick, superficial 
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Figure 4. An fvaludtMm Htm Sent to flectpisnts o1 a Computer-Qenarated fitt^Hography 



EVALUATION/DIALOG BtB 0m 



\. Did the cofoputer bibliography cover the topic requested? 
D yes □ no comroent 

2. Was the turnaround time satisfactory? 

□ yes D no comment 

3« Old you retrieve any Items from the bibliography? 

□ yes □ no 

k. Would a bibliography of 10 citations/abstracts have 
satisfied your request? 

D yes S no 

S« How do you feel about the costs of the service? 
0 reasonable D unreasonable 

6. Do you plan to use the service again? 
D yes C no comments 



M^arch of one or two data banes with no relevance judg* 
mentfi mace on the maximum output of 100 citations or 
ab^traetH. Wliile these evaluation forms may not be fully 
reprcH^ntative. they pnnide an example of the current 
<^trateg>' tor priKiuci/!ier\ice evaluation. 

impaet and Ultttamtiim 

While information centers* especially those operating Ui 
a respimsive mode, tend to represent the "pull** dis* 
semination approach rather than the "push/* externa! 
funding and n*lated accountabiUty issues raise the 
evaluation spectre of measuring impacL What effect 
does an information center and its service have upon 
what happens to students in the classrooms of local 
sct^Kil districts? As with most indirect mource atidca* 
tiiins* the impact of information and its utilization raises 
very dilHcult issues of evaluation strategy. Furthermore, 
the complex factors affecting changes in schools com* 
plicate the cause-and -effect evaluation. One might take 
the pcMiition that v/as recently expressed by a high-level 
state department of education otficial commenting on 
the relative value of state-supported educational infor* 
mation services that school personnel continue in ever 
expanding numbers to make inquiries, request service* 



and use the information center is sufficient justification 
for continued state support. In its follow*up evaluation 
questionnaire of the search product, at least one center 
has difi!Cted inquiries to its users as to the nature and 
value of the information received with questions such as 
those shown in Figure 5« 

It is ditticult to ascertain the retath^e tmpaet of an 
information product on decision-making processes and 
the complex variables that sunound a particular educa* 
tional change. Furthermore, many of the existing centeis 
began and continued theit operation as services rather 
tha^ as change or reform agenci^ Ind^* most centers 
find themselves in a position of having one leg in the 
change role and one in the service function. Especially 
with new clients, the information center tends to concen- 
trate on providing information the client wants* if the 
center Is successful, its credibllhy increase and repeated 
usage by the client occurs. FurthemK)re, the client 
recognizes the value of the service and asks more jaidg- 
mental questions. While most centers prefer not to select 
solutions for their users from their available data base, 
they may provkle technical assistance to help clients 
effectively utilize the information provided. 

In planning a yet uncompleted master evaluation of 
one information center, Taylor (1975) made a distinction 
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Fifturfi 5. FoitowHip EviduAtioA QiwittionMini 



Describe briefly the dctuat u$e you made of the Information or 
assistance provided by the EIC (Educational Information Center). 

Rate the extent to which the information received figured in 
solving the problem which caused the request to be initiated. 



( ) ( ) 

The Information did not 
play a role in solving 
the problem. 



( ) 



( ) ( ) 

The information was the 
major component In rhe 
solution of the problem. 



If you checl«ed "the information did not play a role In solving 
the probl^," then please try to specify below why this was the 
case and do not answer any further questions . 

^^^^ Information was not relevant 

information did not arrive on time 

other resources found were more useful 

the Information need no longer exis , 
other 

Comment: 



(a) Who was affected by the Information or assistance you obtained 
from the EIC? (checic those that apply) 

just myself ...^ student (s) don't l<now 

^^^^ teacher (s) supervlsor(s) no one 

prlncipal(s) parent(s) other (specify^ 



(b) If anyone was affected by the Information or assistance you 
obtained from the EiC, estimate the number of persons affected 
In each category. 

teacher (s) student (s) parent (s) 

principal (s) admlnlstrator(s) other (specify 
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IMPBOVINQ EDUCATION 



betv^'tHfn product ovaluatton and impact evaluation. It 
was presunted that a structured interview eliciting 
respiYn:^e5> from the client on the impact of the Infor-- 
matton services provided might have to deal with impact 
arrayed along five a^^pects: 

1 . the aggregate of all unmeasurable components 

2. new client perceptions of the information service 

3. new client insights into true educational need 

4. changed client attitudes or abilities to deal with 
e?(temal and interna! pressures and constraints 

5. c!ient*initiated decisions and actions which improve 
education (see Figure 6K 

The first aspects that cannot be i^udied are those 
immeasurable, unacknowledged* and elusive compo* 
nents of cHent attitudes and pen:eption$. Part of the true 
value of the service may be found in the modification of 
client personality variables, new perceptions of internal 
strengths and weaknesses* changed outlook upon role* 
clearer assessment of personal abilities to cope with the 
job at hand* and so forth* This aspect of impact may be 
very important, but it is nevertheless difllcuH to assess or 
systematically observe. 

The second aspect of impact has to do with new un- 
derstanding regarding the service and its abilities to help 



the client. This understanding comes about as a result of 
having tested or retested the dissemination system. This 
component is measured by the extent the client is willing 
to report his changing perceptions to the center and the 
degree to which clients make repeat inquests to the 
center for information services. 

The third component of impact at^ the new msights 
the client gains Into his true educational needs. Perhaps 
some aspect of true need, previously unappreciated, is 
now brought to the front atid dealt with as a curreni or 
••felt" need. 

The fourth component of impact Is change attitude 
toward* and increased ability to cope with, the internal 
and external pressures and constraints (e,g.. more confi- 
dence when dealing with issu^ within the community)* 
Sometimes a request for information is basically ne^- 
tive: that is, the need is expressed more in terms of 
coping with a constraint than instituting an effective 
behavior. Most educational information centers are 
willing to respond to such requests te.g., to help educa- 
tors gain materia! to resolve board disputes, make pre- 
sentations to hostile taxpayer groups, and soon). 

The fifth and most important component of impact 
consists of decisions and actions initiated by the client, 
increasing his efiectiven^s and ultimately leading to a 
better education for targeted students. It is. of course* 



ditHcutt Ui establish tht* rciattonship between new educa* 
ttunai practices and the process by which they were instt* 
fitted. It cannot be kncm*n, for example, the extent to 
which information cauiied basic changes in Vocational 
practices. However, the user of systematically dis- 
seminated informatiiin is usually able to assess the extent 
to which he used such information when attempting to 
reach decisions or institute actions. 

From the provision of direct information services to 
the client* through, for example, a collection of abstracts 
or primary documents, a center may move towatti 
pnnision of human interface services* One then en- 
CiHinters an entirely ditierent kind of evaluation 
problem. A major thrust of many information centers 
has been the addition of linker or extension-agent 
functions. While this paper is too short to Include a com* 
plete discussion of this adjunct io information centers* it 
is impiirtanf to delineate at Imt two of the evahiation 
developments which tie these services back into the 
information function itself. Of greatest significance was 
the support by the National Center for Educational Com* 
munication of USOE (later transferred to the National 
Institute of Education) of the eifom of three pilot states 
to establish, maintain, and operate educational Infor- 
mation service centers for a three^year period. These 
ettorts included utilisation of part*time or fulNime 
linkage agents or extension agents. An extremely valua* 
Me evaluation and documentation of the experiences of 
th^i^se three states was completed under the able leader- 
ship of Dr. Sam Sieber of Columbia University's Bureau 
of Applied Social Research. The reports entltl^ Thv Use 
of Educational Knowledge (Sieber 1972K analyzes 
through rather extensive evaluation strategies* key con* 
cerns in the development of a comprehensive informa* 
trim service including the area of management, infor- 
mation retrieval, and Held-agent functions. It is In the 
relationship between the latter two elements (informa- 
tion retrieval and the field-agent function) that Sieber 
conclusions and inferences are most valuable to the 
present discussion of the evaluation of informatbn 
centers. Most important is the transformational role 
provided in some cases by the linkage a^nt — that is* the 
translatiim of the finished product of the information 
center into utilitarian knowledge for use by the client. 
Emphasis ts placed on establishing the neutral role of the 
field agent in this setting rather than the expert pushing 
a particular solution or product on the user. 

Other attempts have been made to measure impact 
utilization but because of problems such as the subtle 
application or influence of information, the high degree 
of information adaptation that takes place* the extreme 
diHiculty in tracing a real communication line, and in the 
absence of appropriate measures, few studies have truly 
been able to determine impact. One promising study was 
developed by Dr. Richard K. Herlig of the Kansas State 
Department of Education. Dr, Herlig developed instru- 



mentation and measurement techniques to evaluate a 
compa^henssve state educatbn information system 
utilising computer search services and afibrding other 
services similar to those oH*ered by a majority of the 
existing information centers, A replication of this study 
was recently conducted by the New York State Depart- 
ment of Education's Education Programs and Studies 
Information Service Unit. !ti summarlEing the sti^tegy 
used, Greg Benson of the New York State Department 
reports as follows: 

The developmental evaluation techniques which were 
designed by Herlig to objectively at^ non-tiK^hnkally 
measure the impact of project services on schools being 
served were based, in part, on Rogers* **sociat systems** 
model. 

Indices were devetoped which incoiporatcd the 
theoretical view thut receptivity to change is controlled by 
(I) the relative complexity of the idea (htformation 
request); (2) the d^ree of innovativeness displayed by the 
individual or grwp initiating the information request; and 
(3) the degree of individual variability in the adoption 
proce^. (Herlig, 1973). 

Working with the above assumption, thiw variables 
were delineated by Herlig: complexity, innovativeness. and 
adoption. These were each viewed as being measurable 
along a continuum of 1 to 5. 

For an efi*ective evaluation, and as t^Hcation of the 
Herlig study in Kansas, it was determhied that for this 
study the three variables be m^sured (complexity, inno^ 
valivene:^ and adoption) for each search. Complexity and 
adoption variables were arrived at through personal inter- 
view and evaluatton form techniques, while innovativeness 
was determined for each educator or group by means of 
the LAIN scale (developed by Herlig in 1971). 

The results of both studies indicate the positive value 
of the use of some kind of informational linking agent 
between the information center itself and the practi- 
tioner in the field, Dbterences between these two studies 
and the general conclusions reached by Sieber and his 
group in regard to the pilot states suggest the need for a 
very careful review of the evaluation methodology and 
continued focus on the elusive impact outcomes of 
educational information services. 

Cmi Eff eetivi»tess 

While the previous section leaves open issues concerning 
current capacities of measuring impact of educational 
information centers, the present accountability syn- 
dn>me raises the other evaluation related concern: cost 
effectiveness. Efforts to analyze the value of Information 
servkes (regardless of the discipline) have been, at best, 
poorly defined. Furthermore, simple marketplace con 
siderations when complicated by the vagaries of grant or 
contract funds from federal or state souttres have ob* 
scored the cost-effective factors. While certahi subcom- 
ponents of the information center's internal operations 
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are subject to comparative analysis (e.g.. one batch com- 
puter search against another type of softwanr system 
given common hardware requirements^ the educational 
information centers have not proliferated suHiciently to 
be subject to the natural selection process frequently 
accorded to the free enterprise system; rather, the 
development of a one*stop comprehensive educational 
information center has been the goal of many infor* 
mation center directors- The information center has 
usually been recreated in a vacuum whet^ there was little 
or no competition to shape or guide its subsequent 
deviflopment. Cost-etfectiveness analysb obviously 
requires an agreed-upon set of cost factors that can be 
sifted out of the overall operations of the information 
center. Weisman has produced one ^eric guide for 
determining the cost of information center operation 
although most information Analysis Centers (which are 
the basis for this document) tend to use expensh^e trans- 
formation and synthesis personnel generally experts in a 
science-l^hnology field. Educational information 
centers seldom provide^ these rol^ and thus, the bench* 
marks are mn fully relevant. Other costing efforts have 
been defined by Douglas Price of the Operation Research 
Corporation {formerly Leasco. a contractor for the ERIC 
facility over the last several years)* but here the emphasis 
is upon the acquisition side rather than the dtssemina- 
tion side of services. While individual operations can 
clearly be defined, particularly those that are labor-in* 
tensive, or where production can be associated with 
direct tabor and materials cost* it is in the elusive area of 
data base acquisition and maintenance, developmental 
activities* and the allocation of overhead costs where cost 
analysis becomes extremely difficult. One present large- 
scale effort at Ci>st*^etfectiveness analysis interestingly 
does not compare one information center style )r 
philosophy against another but instead defines the infor* 
mation center as a linkage agency among a variety of 
linkages within the educational domain. This study Un 
prcHress) is a design for the evaluation of multiple varia- 
tions of educational linkage programs and is being con* 
ducted by the Institute for Communication Research at 
Stanford University. William Paisley and his colleagues 
(1973) have designed an interesting analysis model: 

Under a contract from the National Institute of 
Education. Stanford is conducting a survey of educational 
"linkage** organizations and is designing an evaluation 
procedure that can be used for policy planning by such 
organia^tions and by NIE. 

Educational linkage organisations include state and 
local information centers^ teacher centers, instructional 
materials centers, school study councils. pn)f(E^k>naJ asso- 
ciations. technical assistance programs, college and uni- 
versity education libraries, educational broadcasting for 
educators, and diverse other facilities and programs. 

The linkage evaluation focuses on three factors: 

1. Linkage services, classified generally as retrieval 



s^ervices, publication services* media services, and 
human (interpersonal) services 

2. C^ts of services, classified generally as labor, 
materials* and overhead 

3. Client reactions to services, obtain^ via mail 
questionnaires and person inteniews 

Recommendations and Condttston 

I * Eff orts must be made to bring some consistency to the 
terminology associated with educational information 
centers* their roles, and their evaluation needs. This 
task should be carried out in full cogni7.ance of 
similar efforts emerging from the broader field of 
information scienc*^, The recent establishment of 
Information Services to the Education Special In* 
terest Group within the American Association of 
Information Science should assist this effort. 

2. The notion of ^ucationai information services versus 
the utilization of educational information in the 
e ucational change proems must be more fully 
researched. 

3. More systematic efforts must be made to evaluate 
growing training resoua'es in light of specific opera- 
tional aspects of the educational information center. 
Most Tniportant are the skills related to query nego* 
tiation, search techniques, relevance judgments 
(recall and precision)* and extension agent or other 
utiliEation support. 

4. While nu^re rigorous evaluation strategies are needed 
for impact evaluation of information centers* replica* 
tions of existing studies (such as that of Richard 
Herlig) at more sites are need^ to ascertain their 
applicability to a wide variety of centers* 

5. Existing centers must apply more rigorous program 
budgeting and cost*anaIysis t^hniques in order to 
provide a management data base from which cost* 
benefit characteristics can be determined. 

In conclusion, it is time to take stock of the current 
developments in order to guide and assist new system 
operators in the future. We concur with the commentary 
of Vinsonhaler and Moon U973, 21): 

In summary, much development work has taken place 
during the last five years, to^^her with some evaluation. 
Many types of applications, such as ERIC, are at stages of 
development that now lend themselves to fUrther evalua- 
tion and improvements based upon this evaluation. The 
reviewers feel that, in the future, the success of many types 
of applications will be dependent upon long-term commit- 
ments using evaluation, re^analysls of goals, modification 
of the system, and continued evaluation in a repetitive 
process. 

Further, in coming years the need seems to be less for 
additional technical developments than for persons capa* 
ble of relating the existing technical tools to the ver> 
human process that is education. 
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